System and method for presentation and control of augmented vehicle surround views

ABSTRACT

Methods and systems for presentation and control of an augmented virtual vehicle view with surround view imaging, that include generating a stitched image view based on surround view image data for a vehicle and generating a pose estimation for the vehicle, wherein the pose estimation provides a vehicle position and a vehicle orientation relative to the stitched image view. In one or more examples, the methods and systems include augmenting the stitched image view to include one or more graphical elements based on the pose estimation for the vehicle, presenting a graphical display of the stitched image view including the one or more graphical elements and a vehicle avatar. Further, in at least one example, the methods and systems may include detecting position commands for a position of the vehicle avatar within the graphical display and updating the avatar position and iconography in response to the position commands.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 15/582,488, entitled “SYSTEM AND METHOD FORPRESENTATION AND CONTROL OF AUGMENTED VEHICLE SURROUND VIEWS”, and filedon Apr. 28, 2017. The entire contents of the above-listed applicationare hereby incorporated by reference for all purposes.

FIELD

The present disclosure relates to systems, methods and devices forpresentation of surround view imaging, and more particularly topresentation and control of augmented vehicle virtual views withsurround view imaging.

BACKGROUND

In recent years, vehicles have been outfitted with cameras to provide anexternal view. Many vehicles include single camera back-up cameras forparking assistance. In addition, the inclusion of cameras for vehicleshas increased. Often these cameras are used for driver assistance and toimprove vehicle safety. While cameras are known to provide views forparking assistance, use of these devices is often limited. In addition,many vehicles disable operation of cameras to prevent driver distractionwhile a vehicle is in motion. There exists a need to utilize vehiclecameras to provide additional functions. There also exists a need toaugment image data for presentation in the vehicle.

BRIEF SUMMARY OF THE EMBODIMENTS

Disclosed and claimed herein are methods, devices and systems forpresentation and control of an augmented virtual vehicle view withsurround view imaging. One embodiment is directed to a method includinggenerating, by a first device, a stitched image view based on surroundview image data for a vehicle, wherein the surround view image data isgenerated based on multiple image sensors of the vehicle. The methodalso includes outputting, by the first device, the stitched image viewand virtual camera position data to an augmented reality module of thefirst device. The method also includes determining, by the first device,a pose estimation for the vehicle, wherein the pose estimation providesvehicle position and orientation relative to the stitched image view.The method also includes augmenting, by the first device, the stitchedimage view to include one or more graphical elements based on the poseestimation for the vehicle and virtual camera position. The method alsoincludes presenting, by the first device, a graphical display of thestitched image view including the one or more graphical elements.

In one embodiment, the surround view image data includes image data fora complete 360 degree circumferential image with respect to position ofthe vehicle and the stitched image relates to a portion of thecircumferential image from the perspective of the virtual cameraposition.

In one embodiment, a surround view module of the first device outputsthe stitched image view and virtual camera position data to an augmentedreality module of the first device, the virtual camera position dataincluding six degree of freedom camera data.

In one embodiment, an augmented reality module of the first devicedetermines a pose estimation for the vehicle based on data generated bya vehicle sensor for acceleration and tilt of the vehicle.

In one embodiment, augmenting the stitched image view by the firstdevice includes presentation of a vehicle avatar and iconography forpoints of interest based on the pose estimation for the vehicle andvirtual camera position.

In one embodiment, presenting the graphical display of the stitchedimage view including the one or more graphical elements includesarranging iconography for one or more points of interest in a displaywindow based on the pose estimation, wherein the pose estimation andvirtual camera position allow for placement of the iconography intostitched image based on the direction of the vehicle.

In one embodiment, virtual camera position data provides six degrees offreedom data characterizing the stitched image view output by a surroundview module.

In one embodiment, augmenting the stitched image includes visualizationof the points of interest in the surround view image data providespresentation of graphical elements to match physical locations ofelements outside of the vehicle.

In one embodiment, the method further includes detecting changes to avirtual camera position and updating the output image to modify positionof graphical elements in a displayed view.

In one embodiment, the method further includes detecting positioncommands for the avatar position within the output display and updatingposition of the avatar and iconography.

Another embodiment is directed to a vehicle control unit forpresentation and control of an augmented virtual vehicle view withsurround view imaging. In one embodiment, the vehicle control unitincludes a surround view module configured to generate a stitched imageview based on surround view image data for a vehicle, wherein thesurround view image data is generated based on multiple image sensors ofthe vehicle, and output the stitched image view and virtual cameraposition data to an augmented reality module of the first device. Thedevice also includes an augmented reality module configured to determinea pose estimation for the vehicle, wherein the pose estimation providesvehicle position and orientation relative to the stitched image view,and augment the stitched image view to include one or more graphicalelements based on the pose estimation for the vehicle and virtual cameraposition. The device also includes a control input module configured tocontrol presentation of a graphical display of the stitched image viewincluding the one or more graphical elements.

Other aspects, features, and techniques will be apparent to one skilledin the relevant art in view of the following detailed description of theembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, objects, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify correspondingly throughout and wherein:

FIG. 1 depicts a graphical representation of components for presentationand control of an augmented virtual vehicle view with surround viewimaging according to one or more embodiments;

FIG. 2 depicts a process for presenting an augmented virtual vehicleview according to one or more embodiments;

FIG. 3 depicts a graphical representation of device components accordingto one or more embodiments;

FIG. 4 depicts a graphical representation of virtual vehicle viewsaccording to one or more other embodiments;

FIGS. 5A-5D depict system diagrams for augmented virtual vehicle viewwith surround view imaging according to one or more embodiments;

FIG. 6 depicts a graphical representation of virtual camera positionaccording to one or more embodiments;

FIG. 7 depicts a graphical representation of virtual camera positionaccording to one or more other embodiments; and

FIG. 8 depicts a system diagram according to one or more embodiments.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS Overview andTerminology

One aspect of the disclosure is directed to providing surround viewimage data for a vehicle and augmenting the surround view. Surround viewimaging relates to generating image data to provide a complete, 360degree, image of vehicle surroundings. Embodiments discussed hereininclude the use of multiple cameras to capture multiple views from thevehicle and a surround view module to generate a surround view. Thesurround view module may output a portion of the surround view. Mostdisplays are two dimensional and the stitched image allows for displayof a portion of the surround view, even if image data of the stitchedimage is from multiple image detection sources. According to theprinciples discussed herein, control inputs may be detected and utilizedto modify the portion of the surround view presented. Image data outputfor display may be stitched image data generated as a composite of imagedata detected by the multiple camera inputs.

Embodiments are directed to presentation and control of an augmentedvirtual vehicle view with surround view imaging. System and deviceconfigurations are provided to generate a surround view image, receivecontrol inputs, augment surround view image data and output augmentedimage data relative to a vehicle including graphical elements such asiconography. In one embodiment, the augmented surround view image datais provided for passenger entertainment and use to allow for display ofpoints of interest. According to another embodiment, augmentation allowsfor gaming elements to be integrated with surround view image data.

According to another embodiment, stitched image data may be presentedfrom the perspective of a virtual camera position. In certainembodiments, the stitched image is coincidental with the virtual cameraposition. A virtual camera position may relate to a particular viewpoint for representation, such as from the perspective of above avehicle and/or behind the vehicle looking out in the direction of travelor other directions for that matter. A virtual camera position does notrelate to an actual position of an image sensing device, but rather arepresentation from a vantage point for projecting image data. Accordingto one embodiment, the surround view image relates to an inner toroidsurface wherein the stitched image represents the inner wall of thetoroid, the inner wall generated by stitching image data from themultiple camera inputs together. The stitched image may relate to anaugmented reality as image data presented relates to composite imagedata generated from one or more image sources.

Embodiments discussed herein are directed to augmentation of stitchedimage data by presentation of graphical elements in a three dimensionalspace using virtual camera position data. In addition to utilizing avirtual camera position, virtual camera position data may be generatedand utilized to determine wherein in three dimensional space graphicalelements should appear. The virtual camera position data may describethe perspective that stitched image data is taken from using six degreesof freedom data. The virtual camera position data my include outputtinga coordinate space generated with the stitched image. In addition tovirtual camera data, pose estimation of the vehicle may be performed toidentify vehicle arrangement. The pose estimation may provide vehicleposition and orientation relative to the stitched image view. In oneembodiment, augmentation of stitched images include overlaying one ormore graphical elements based on the pose estimation for the vehicle andvirtual camera position.

In one embodiment, system and device configurations are providedincluding a control module, surround view module and augmented realitymodule to allow for augmentation of surround view image data for avehicle while allowing for manipulation of a virtual camera view.

According to another aspect, surround view images for a vehicle may beaugmented with gaming elements. System configurations, devices andmethods are discussed herein for augmentation surround view images withgaming system iconography and control of display elements based oncontrol inputs.

Another aspect of the disclosure is directed to providing a passengerdisplay system with control to allow for rotation and control of adisplay view and thus, presentation of iconography in a threedimensional space independent of surround view image detection. Thepassenger display system may include generating and orienting virtualiconography in video merged together from multiple sources. The displaysystem may include determining the viewport from external input andproviding the window to an augmented reality overlay engine to augmentsurround view image data. The device may include a blockchain apparatusfor “in-app” purchases, a point of interest (POI) interface for genericPOI injection, use with tour guides, gaming icons, etc.

Another aspect of the disclosure is directed to augmented reality usingmultiple camera inputs. Virtual iconography may be generated andoriented in video merged together from multiple sources. Waypoints andPOIs may be customized and interacted with during vehicle operation.

As used herein, the terms “a” or “an” shall mean one or more than one.The term “plurality” shall mean two or more than two. The term “another”is defined as a second or more. The terms “including” and/or “having”are open ended (e.g., comprising). The term “or” as used herein is to beinterpreted as inclusive or meaning any one or any combination.Therefore, “A, B or C” means “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C”. An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

Reference throughout this document to “one embodiment,” “certainembodiments,” “an embodiment,” or similar term means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment. Thus, the appearancesof such phrases in various places throughout this specification are notnecessarily all referring to the same embodiment. Furthermore, theparticular features, structures, or characteristics may be combined inany suitable manner on one or more embodiments without limitation.

Exemplary Embodiments

Referring now to the figures, FIG. 1 depicts a graphical representationof components for presentation and control of an augmented virtualvehicle view with surround view imaging according to one or moreembodiments. System 100 may be configured to output a stitched imagerelative to a virtual camera position, input controls and one or moregraphical elements to identify points of interest and iconography.According to one embodiment, system 100 relates to a vehicle controlunit. System 100 includes a control input module 105, surround viewmodule 110 and augmented reality module 115. System 100 may additionallyinclude elements and modules as discussed herein. According to oneembodiment, a control input module 105, surround view module 110 andaugmented reality module 115 may each relate to one or more of aphysical module, such as controller (e.g., processor, processing unit,etc.), and software module, such as an application or program within avehicle unit. As will be described below, control input module 105,surround view module 110 and augmented reality module 115 may beassociated with one or more devices. By providing virtual cameraposition data, augmented reality module 115 is provided with a directionand coordinate mapping for the stitched image in space relative to thelocation of the vehicle. By then estimating vehicle attributes, such asa pose estimation, object locations may be positioned in space relativeto the virtual camera position data.

According to one embodiment, control input module 105 controls bothsurround view module 110 and augmented reality view 115. According toanother embodiment, control input module 110 may be configured tocontrol presentation of augmented imagery relative to a vehicle. In oneembodiment, control input module 105 outputs a command to surround viewmodule 110 and augmented reality module 115 to present a stitched image.The initial presentation of the stitched image may relate to a forwarddirection of a vehicle. Based on one or more use commands detected bycontrol input module 105, commands may be output to surround view module110 and augmented reality module 115 to modify the viewing angle, updatethe presentation of display elements and control presentation output.Control input module 105 may include full control of virtual cameraposition.

According to one embodiment, surround view module 110 is configured toreceive image data 120 and output a stitched image, shown as 125, andvirtual camera position data, shown as 130, to augmented reality module115.

According to one embodiment, input image data 120 relates to image datacaptured by a vehicle, such as multiple camera inputs relating to afront, left side, right side and rear view of the vehicle. Surround viewmodule 110 is configured to generate a surround view image and output astitched image 125. In one embodiment, surround view module 110generates a surround view image by combining image data detected bymultiple cameras. Surround view module 110 may then combine the imagedata and output stitched image 125 representing a portion of thesurround view. In certain embodiments, stitched image 125 relates to aportion of the surround view associated with a virtual camera position.In other embodiments, stitched image 125 relates to a portion of thesurround view image data selected to present on a display of the vehiclebased on input detected by control input module 105. Virtual cameraposition data 130 provides six degrees of freedom data associated withstitched image 125. By way of example, virtual camera position data 130provides each of a forward/back data set, up/down data set, left/rightdata set, pitch data set, yaw data set, and roll data set associateddescribing the direction of a virtual camera position.

According to one embodiment, augmented reality module 115 is configuredto receive a stitched image 125, virtual camera position data 130 andcontrol input commands to output an augmented image output 140.According to another embodiment, augmentation of image data such assurround view image data may include output of additional elements suchas sound based on one or more of a detected pose, presented view, userinput, vehicle location, vehicle operation, etc.

FIG. 2 depicts a process for presenting an augmented virtual vehicleview according to one or more embodiments. Process 200 may be employedby a device, such as a vehicle control unit of a system (e.g., system100) and one or more other components to output a surround viewpresentation. Process 200 may be performed by a single device ormultiple devices. As will be described herein, a surround view moduleand augmented reality module of the first device may be software modulesexecuted by a processor. In other embodiments, the surround view moduleand augmented reality module may be performed by two devices, such as avehicle unit and module bile interoperating with the vehicle unit.

According to one embodiment, process 200 may be initiated by a firstdevice or module (e.g., surround view module 110) generating a stitchedimage view based on surround view image data for a vehicle at block 205.In one embodiment, the surround view data is generated based on multipleimage sensors of the vehicle. For example, the surround view image datacan include image data for a complete 360 degree circumferential imagewith respect to position of the vehicle and the stitched image relatesto a portion of the circumferential image from the perspective of thevirtual camera position. The stitched image may be generated based on avirtual camera position. For example, a control input module or userinput can indicate a desired viewing window. In response, the stitchedimage may be generated form surround view image data to match theviewing angle and direction of the virtual camera position data. Thevehicle system and surround view module, in particular, may generatevirtual camera position data to indicate the viewing angle and directionof the virtual camera position. In one embodiment, virtual cameraposition data provides six degrees of freedom data characterizing thestitched image view output by a surround view module. By providingvirtual camera position data to an augmented reality module, theposition of graphical elements can be matched to a 3-dimensional space.In one embodiment, a visualization positions iconography to be within a3 dimensional coordinate system as an overlay to image data. In thatfashion, the system does not have to parse the actual image data itselfto determine location for presentation of items, location of items canbe updated as the virtual camera position viewing direction changes, anditems can be placed using the virtual camera position data. In someinstances, based on changes in viewing direction, items can be removedfrom view when a viewing direction or field of view is no longerassociated with position of a point of interest.

At block 210, the stitched image view and virtual camera position dataare output to an augmented reality module of the first device. In oneembodiment, a surround view module of the first device outputs thestitched image view and virtual camera position data to an augmentedreality module of the first device, the virtual camera position dataincluding six degree of freedom camera data.

At block 215, a pose estimation is determined for the vehicle. In oneembodiment, an augmented reality module of the first device determines apose estimation for the vehicle based on data generated by a vehiclesensor for acceleration and tilt of the vehicle.

At block 220, the stitched image view is augmented to include one ormore graphical elements based on the pose estimation for the vehicle andvirtual camera position. In one embodiment, augmenting the stitchedimage view by the first device includes presentation of a vehicle avatarand iconography for points of interest based on the pose estimation forthe vehicle and virtual camera position. According to anotherembodiment, augmenting the stitched image includes visualization of thepoints of interest in the surround view image data provides presentationof graphical elements to match physical locations of elements outside ofthe vehicle.

At block 225, a graphical display of the stitched image view ispresented to include the one or more graphical elements. In oneembodiment, presenting the graphical display of the stitched image viewincluding the one or more graphical elements includes arrangingiconography for one or more points of interest in a display window basedon the pose estimation, wherein the pose estimation and virtual cameraposition allow for placement of the iconography into stitched imagebased on the direction of the vehicle. Iconography such as graphicalelements for a point of interest, gaming characters, gaming targets,gaming obstacles, icons and alerts may be positioned and presented by adisplay with stitched image data. In one embodiment, object presentationis determined based on vehicle position. By way of example, as a vehiclenears a point of interest, the graphical element representing the pointof interest may be enlarged and/or the display position may be modified.In gaming applications, iconography and gaming elements may bemanipulated. For example, a vehicle avatar may be manipulated based oninput controls to change the direction and even manipulate displayposition to direct the avatar towards one or more gaming targets (oravoid gaming elements displayed), in which case the target positioningmay be based on virtual camera position data and control commands fordisplayed and controllable elements.

At block 225, output of image data and augmented display elements mayinclude output of one or more audio elements. For example, when a viewis modified to present a point of interest one or more sound effects maybe output to notify a user of a graphical element in a displaypresentation. For gaming applications, sound output based on the virtualview may be started, modified or stopped in association with user inputcommands and display elements depicted. Audio elements output by thesystem may include alerts, music, recorded voice, driver assistance,etc.

In one embodiment, process 200 includes detecting changes to a virtualcamera position and updating the output image to modify position ofgraphical elements in a displayed view. An input device such as acontroller or touch screen interface of a display interface may detectcommands to change or rotate viewing angle for the virtual camera. Thepresentation may be updated to reflect detected input commands and astitched image or portion of the surround view may be presentedassociated with the control input.

In one embodiment, process 200 includes detecting position commands forthe avatar position within the output display and updating position ofthe avatar and iconography. In certain embodiments, control inputs maybe detected resulting in modifying the presentation format of aniconography. For example directional changes with respect to a vehicleavatar may modify the display to show the vehicle avatar as turningrotating associated with the direction change. Once the directionalcommand is released the presentation of the vehicle avatar may return toan initial display format.

FIG. 3 depicts a graphical representation of device components accordingto one or more embodiments. According to one embodiment, device 300relates to a vehicle device configured to provide augmented surroundview image data. Vehicle devices may include audio/video units,infotainment units, display units, media processers, etc. According toone embodiment, device 300 includes processor 305, sensors 310, datastorage unit 315, input/output module 320, and display 325.

Processor 305 is configured to control operation of device 300.According to one embodiment, processor 305 may be configured to run asurround view module 306 and augmented reality module 307. In otherembodiments, surround view module 306 and augmented reality module 307may be physical hardware units of device 300.

In one embodiment, surround view module 306 is configured to generate astitched image view based on surround view image data for a vehicle. Thesurround view data is generated based on sensors 310, which may includemultiple image sensors. Surround view module 306 outputs the stitchedimage view and virtual camera position data to augmented reality module307. In one embodiment, augmented reality module 307 determines a poseestimation for a vehicle to provide vehicle position and orientationrelative to the stitched image view. The augmented reality module 307augments the stitched image view to include one or more graphicalelements based on the pose estimation for the vehicle and virtual cameraposition. Processor 305 may also function as a control input module tocontrol presentation of a graphical display of the stitched image viewincluding the one or more graphical elements.

Processor 305 may include executable code to perform and controlfunctions of device 300. By way of example, processor 305 may executeprocess 200 of FIG. 2 to output augmented image data. Processor 305 mayexecute and direct one or more processes and functional blocks describedherein to present surround view image data and control presentation ofelements. Processor 305 may be a vehicle control unit for presentationand control of an augmented virtual vehicle view with surround viewimaging.

Sensors 310 may be configured detect image data. In one embodiment,sensors 310 include multiple camera inputs configured to detect imagedata relative to the surroundings of a vehicle. According to anotherembodiment, sensors 310 may include acceleration and orientation sensorsfor a device, such as a gyroscopic sensor and accelerometer. In otherembodiments, sensors 310 may include elements to detect globalpositioning, vehicle speed, operability, external condition (e.g.,distance to target, etc.) or other vehicle applications. Sensors 310 maybe configured to provide output of the sensors to processor 305. Incertain embodiments, data storage module 315 may store output of sensors310, and/or processor 305 may direct or control operation of sensors310.

Data storage unit 315 may be configured to store executable code tooperate processor 305 and device 300. Input/output (I/O) module 320 maybe configured to receive inputs from a controller or input surface(e.g., touch screen, input buttons, etc.) and to output image data todisplay 325. In certain embodiments, device 300 includes is configuredto communicate over a communication network 330 (e.g., IP network) byway of I/O module 320. Device 300 may optionally request and/or outputdata to optional data server 340 by way of communication network 330.

In certain embodiments, device 300 and processor 305 may be configuredto communicate with components of a vehicle, such as vehicle system 335.By way of example, vehicle system 335 may relate to a user interfacesystem of a vehicle including one or more sensors, functions and datacapabilities. In that fashion, vehicle system 335 may provide one ormore of location data, and vehicle operating data to processor 305,including but not limited to GPS data, heading, acceleration, mapinformation, points of interest, etc.

FIG. 4 depicts a graphical representation of virtual vehicle viewsaccording to one or more other embodiments. According to one embodiment,surround view system 400 for vehicle 401 is depicted with exemplaryaugmented representations 405 and 410. According to one embodiment,surround view system 400 includes a plurality of cameras configured todetect surroundings 407 _(1-n), including front surrounding 407 ₁, sidesurroundings 407 ₂₋₃ and back surrounding 407 _(n). In certainembodiments, system 400 includes sensors for front diagonal surroundings408 ₁₋₂ and diagonal back surrounding 408 _(3-n). According to oneembodiment, surround view system 400 is configured to detect a surroundview image for vehicle 401, includes image data to provide a complete360 degree representation of vehicle surroundings. In addition, surroundview system 400 is configured output a stitched image based on a virtualcamera position of vehicle 401. In one embodiment, for example, thevirtual camera position may relate to the top 402 of vehicle 401 lookingout in any direction from the vantage point. Accordingly, the stitchedimage output by system 400 may be generated from the vantage point ofposition 402, even though no camera is actually placed at position 402of vehicle 401. Surroundings 407 _(1-n) and surroundings 408 _(1-n) maybe detected by one or more cameras each, where the cameras may includefish eye or other lens to broaden the viewing angle.

According to one embodiment, augmented representation 405 relates to astitched image 415. According to another embodiment, an augmentedreality system of system 400 augments stitched image 415 by includingavatar 420 and one or more graphical elements 425 and 430 representingiconography for points of interest. Vehicle avatar 420 may be selectedby a user and/or modified by the system. Points of interest 425 and 430may relate to one or more of services, address book locations,landmarks, map items, etc. According to one embodiment, the position ofpoints of interest 425 and 430 may be based on virtual camera positiondata received by an augmented reality module to position points ofinterest 425 and 430 in a three-dimensional space and thus overlaystitched image 415. By positioning points of interest 425 and 430 usingvirtual camera position data, the system can modify the position ofelements an update views as a vehicle moves and as inputs to the devicerotate and/or change the virtual camera position or viewing direction.

Augmented representation 410 relates to a combination of stitched image435 presented on a display with map data 440. According to oneembodiment, an augmented reality system of system 400 augments stitchedimage 435 by including avatar 420 and one or more graphical elements 450and 455 representing points of interest. Map data 440 includes vehicleposition represented by a graphical element 445 relate to map data andposition of a point of interest 450. According to one embodiment, one ormore of stitched image 435, map data 440 and points of interest 450 and455 may be based on virtual camera position data and a pose estimationof the vehicle.

Augmented representation 405 and augmented presentation 410 eachrepresent virtual views or a representation of the surroundings ofvehicle 401. According to one embodiment, augmented representation 405and 410 may be presented by a display (e.g., display 325) in a vehicleduring operation of a vehicle. Virtual camera position may use imagedata detected by the vehicle sensors, however, data from each imagesensor may be combined, such that a virtual view is provided from one ormore imaging devices.

FIGS. 5A-5D depict system diagrams for augmented virtual vehicle viewwith surround view imaging according to one or more embodiments.According to one embodiment, systems in FIGS. 5A-5D are configured togenerate augmented virtual vehicle views. Referring to FIG. 5A, system500 includes device 505 configured to output augmented surround viewimagery. Device 505 may receive data from one or more sources for avehicle. For example, device 505 is connected to and receives image datafrom multiple camera inputs 506, wherein the camera inputs 506 mayinclude separate cameras such as a front view camera (FVC), side viewcamera left (SVL), side view camera right (SVR), and rear view camera(RVC). In one embodiment, one or more augmented reality settings 507 maybe provided to device 505 such as a point of interest filter to identifythe which points of interest to display, avatars for the vehicle,graphic display elements for the points of interest, distanceinformation, etc. Augmented reality settings 507 may includemerchandising, purchasing over the air information, block chainsettings, etc. In addition, one or more external sensors 508 may providedata to device 505 such as a positioning sensor (GPS sensor), etc.

According to one embodiment, device 505 includes a controller inputmodule 510, surround view software module 515 and augmented realitysoftware module 520. Controller input module 510 may control each ofsurround view software module 515 and augmented reality software module520, detect input commands, and set the virtual camera position. In oneembodiment, input commands may be entered to display 525, such as touchcommands, which are provided to controller input module 510.

Surround view module 515 includes virtual camera position module 516 andview generation module 517. According to one embodiment, virtual cameraposition module 516 receives a desired virtual camera position fromcontroller input module 510 and any changes to the virtual cameraposition. Based on the virtual camera position and image data receivedfrom multiple camera inputs 506, view generation module 517 generates astitched image 518 output to augmented reality software module 520.Surround view module 515 also outputs virtual camera position data 519including six degrees of freedom information for the viewing angle, anddirection of the virtual camera position. Virtual camera position data519 is output to augmented reality software module 520.

Augmented software module 520 is configured to present one or moregraphical elements, such as points of interest, avatars, gaming elementsetc., on stitched image 518 received from surround view software module515. According to one embodiment, augmented software module 520 performsa pose estimation of a vehicle and virtual camera position data 519 todetermine a three dimensional space of graphical elements. According toanother embodiment, augmented reality software module 520 receives inputfrom controller input module 510 indicating user input commands toposition graphical elements augmented to the stitched image.

Augmented reality software module 520 includes a virtual gimbal 523which utilizes virtual camera position data 519 to determine ahorizontal plane as a point of reference. Augmented reality softwaremodule 520 also receives vehicle acceleration, direction and angleinformation from gyro and acceleration sensor 524. Pose estimation 521of augmented reality software module 520 utilizes the horizontal planeand vehicle acceleration data to estimate the current position of avehicle and operating characteristics relative to stitched image 518.According to one embodiment, visualization module 522 determinesplacement of graphical elements as overlays and outputs combined view526 to display 525. According to one embodiment, points of interestavailable may be provided by a point of interest (POI) database 530. POIdatabase 530 may be part of a vehicle system.

Display 525 relates to a display unit of the vehicle. In certainembodiments display 525 may be the main display unit of a vehicleinfotainment system. According to another embodiment, display 525 mayrelate to a passenger display unit of the vehicle.

According to one embodiment, augmented software module 520 and/orelements of system 500 may be configured to output sound associated withinput commands or position of a vehicles relative to iconographypresented. With respect to navigation and points of interest, detectedviewing angle changes for the virtual camera position may be associatedwith one or more sounds such as audio feedback. Audio feedback may bedirectional, such that a viewing angle associated with a particulardirection (e.g., vehicle left side) may result in audible output. Inother embodiments, such as a gaming application manipulation of thevehicle avatar, selection of iconography, avoidance (or engagement) withtargets can result in output of one or more sound effects. Output ofsound effects may be directional, such that the sound effects may differbased on the stitched image view presented. In that fashion, virtualcamera position data may be employed to active audio of a first soundwith a particular viewing angle and output of a second or other soundassociated with a second viewing angle and/or vehicle position.

Referring to FIG. 5B, system 535 includes device 505 configured tooutput augmented surround view imagery including one or more gamingelements. System 535 may operate and include similar elements to system500 described above. In FIG. 5B, device 505 is configured to receivedata including game settings 536 provided to device 505 includingcontroller response information, a vehicle avatar/icon selection, etc.Game settings 536 may include full control of virtual camera positionfor use with racing games, and vertically scrolling games. Game settings536 may also include a virtual vehicle database interface for switchingbetween avatar and image elements.

According to another embodiment, controller input module 510 isconfigured to received gaming controls from controller 537. Controller537 may relate to a game controller providing directional commands, oneor hard buttons, etc. In certain embodiments, controller 537 is a sixaxis controller. Controller input module 510 is configured to receiveinputs from controller 537 directed to moving objects displayed onstitched image data 518. In one embodiment, the vehicle avatar augmentedto stitched image data 518 may be manipulated in the display view toallow for a gaming application. By way of further example, one or moregaming elements augmented with stitched image data 518 and output asaugmented image data with iconography at 539 to display 525 may move orbe positioned on the display 525 based on local gaming data 538 andnetwork gaming data 530. In contrast to simply presenting gamingelements on a display, augmented reality engine may position gamingelements on stitched image data 518 relative to a surround view of thevehicle and virtual camera position data.

Referring to FIG. 5C, system 550 includes surround view module 515 andaugmented reality software module 520 on two different devices.According to one embodiment, system 550 includes head unit 551configured to operated with device 552 which may be a mobile device ortablet. Similar to system 535, system 550 may provide gamingfunctionality. It should also be appreciated that the two devicearrangement of FIG. 5C may be applied similarly to the embodimentsdiscussed above in FIG. 5A.

According to one embodiment, head unit 551 relates to a vehiclecomponent, such as a vehicle head unit, infotainment system etc. Headunit 551 may include controller input module 510 and surround viewsoftware module 515. Head unit 551 may also receive game settings 536.Based on image data received from multiple camera inputs 506, viewgeneration module 517 generates a stitched image 518 output to augmentedreality software module 520 of device 552. According to anotherembodiment, controller input module 510 receives input command from oneor more of display 525 and game controller 537. In certain embodiments,head unit 551 and device 552 are connected by a network connection,which may be wired or wireless. Connection formats between head unit 551and device 552 can include audio video connections such as one or moreof an HDMI, DP, USB and Ethernet.

Referring to FIG. 5D, a system configuration is presented according toanother embodiment. System 555 is configured to provide an augmentedsurround view according to one or more embodiments. System 555 includessimilar elements as system 500 of FIG. 5A and system 550 of FIG. 5C. Byway of example, system 555 includes head unit 556 and mobiledevice/tablet 557 similar to the two device configuration of system 550of FIG. 5C. According to one embodiment, a surround view module, such assurround view module 558, is configured to include a visualizationmodule, such as visualization module 560, to determine placement ofgraphical elements as overlays and for augmenting surround view imagedata.

Similar to systems discussed above (e.g., system 500, system 535, system550, etc.) system 555 may provide an augmented surround view which maybe applied to one or more of vehicle displays, point of interestfunctionality and gaming functionality. It should also be appreciatedthat elements of system 555 operate and/or provide functionality similarto similarly numbered elements above.

According to one embodiment, head unit 556 relates to a vehiclecomponent, such as a vehicle head unit, infotainment system, vehicleunit in general, etc. Head unit 556 may include controller input module510 and surround view software module 558. Surround view module 558includes virtual camera position model module 516, view generationmodule 517, and visualization module 560. According to one embodiment,virtual camera position module 516 receives a desired virtual cameraposition from controller input module 510 and any changes to the virtualcamera position.

Controller input module 510 is configured to receive one or more controlsignals from a controller, such as controller 537, or input commandsrelative to an interactive display 563 of mobile device 557. In oneembodiment, display 563 may be a touch screen display and commandsprovided to controller input module may be based on an application oruser interface for an augmented surround view. Controller input module510 is configured to provide a desired view and changes to a particulardisplay configuration to augmented reality engine 559. Based on imagedata received from multiple camera inputs 506, virtual camera positionmodel module 516 provides virtual camera position data 519 to augmentedreality software module 559.

According to one embodiment, pose estimation module 521 of augmentedreality module 559 performs a pose estimation of a vehicle and virtualcamera position data 519 to determine a three dimensional space ofgraphical elements for positioning one or more graphical elements in athree-dimensional space, including but not limited to points ofinterest, gaming elements, navigation elements, etc. According toanother embodiment, augmented reality software module 520 receives inputfrom controller input module 510 indicating user input commands toposition graphical elements. Pose estimation module 521 of augmentedreality module 559 outputs spatial display information 561, such asthree dimensional display information for one more elements, tovisualization module 560. In one embodiment, visualization module 560inputs graphical elements of the augmented reality module 559 into asurround view image using an estimated pose. View generation module 517may select a portion of the augmented surround view for presentation ondisplay 563. In one embodiment, settings 562 allow for user settings ofcharacters, display elements, display attributes and formatting ofelements augmented to the surround view. In a gaming application,settings 562 may include an avatar to be included with the presentation.Settings 562 may provide controller response and vehicle iconselections. Similarly, points of interest, gaming display elements andnavigation elements may be received from one or more of a local database563 and cloud based database 564.

View generation module 517 outputes a stitched image 565 output todisplay 563 of device 557. According to another embodiment, controllerinput module 510 receives input commands from one or more of display525, which may be a touch screen display, and game controller 537.Display 525 may be an integrated touch screen, such that touch commands(e.g., swipe, slide, tap, etc.) can be used to change virtual cameraposition and/or vehicle avatar control.

FIG. 6 depicts a graphical representation of virtual camera positionaccording to one or more embodiments. According to one embodiment,virtual camera position may relate to an initial virtual cameraposition, such as a starting virtual camera position when an augmentedvirtual vehicle view is initiated or based on inputs to the controllerinput module that modify or change the virtual camera position. FIG. 6illustrates an exemplary representation of virtual camera position.Virtual camera position 600 is depicted relative to a vehicle 605 withsurround view image 610, wherein direction 611 indicates the virtualcamera position view and direction associated with a stitched image ofvirtual camera position 600. According to one embodiment, virtual cameraposition 600 relates to a perspective above the roof view of vehicle605. According to one embodiment, a graphical representation 601 ofvirtual camera position includes stitched image 615 representing aportion of surround view image 610 associated with direction 611.According to one embodiment, an augmented reality module augmentsstitched image 615 to include vehicle avatar 616 and graphical elementsfor points of interest 617 and 618. According to another embodiment,virtual camera position data provided to the augmented reality module bya surround view software module and pose estimation for vehicle 605 areused to determine the position of points of interest 617 and 618.Graphical representation 601 also includes one or more elements, shownas 619 (e.g., parking spaces) included in stitched image 615.

According to one embodiment, input commands to rotate and/or translateone or more of vehicle avatar 616 and virtual camera position 600 aredetected by the augmented virtual vehicle view with surround viewimaging and as result include updating of both stitched image data anddi splay characteristics of augmented elements.

FIG. 6 illustrates an exemplary representation of an updated virtualcamera position 620 depicted relative to a vehicle 605 with surroundview image 610, wherein direction 621 indicates the virtual cameraposition view and direction associated with a stitched image of updatedvirtual camera position 620. According to one embodiment, updatedvirtual camera position 620 also relates to an above the roof view ofvehicle 605, and is a left side view of vehicle 605. According to oneembodiment, a graphical representation 621 of the updated virtual cameraposition includes stitched image 625 representing a portion of surroundview image 610 associated with direction 622 (e.g., the left sidesurround view). According to one embodiment, an augmented reality moduleaugments stitched image 625 to include vehicle avatar 616 and graphicalelements for point of interest 617. According to another embodiment,virtual camera position data provided to the augmented reality module bya surround view software module and pose estimation for vehicle 605 areused to determine the position of point of interest 617. Graphicalrepresentation 621 also includes one or more elements, shown as 619(e.g., parking spaces) included in stitched image 615. Accordingly, achange in the virtual camera position detected by a controller inputmodule allows for manipulation such as rotation or translation of thestitched image view and still allows for points of interest to bedisplayed, however, points of interest that are not associated with thewindow of an updated virtual camera position, such as point of interest618 may not be included, while the position of a point of interest, suchas point of interest 617 are retained and repositioned. According toanother embodiment, augmented stitched image 625 may also include agraphical element, such as element 630, display to allow formanipulation or control of the virtual camera position or view.

FIG. 7 depicts a graphical representation of virtual camera positionaccording to one or more other embodiments. According to one embodiment,virtual camera and augmented elements may be associated with a gamingapplication. FIG. 7 depicts an augmented image with manipulation ofelements. FIG. 7 illustrates virtual camera position 700 depictedrelative to a vehicle 705 with surround view image 710, whereindirection 711 indicates the virtual camera position view and directionassociated with a stitched image of virtual camera position 700.According to one embodiment, virtual camera position 700 relates to anabove the roof view of vehicle 705. According to another embodiment, agraphical representation 701 of virtual camera position includesstitched image 715 representing a portion of surround view image 710associated with direction 711. According to one embodiment, an augmentedreality module augments stitched image 715 to include vehicle avatar 716and iconography for gaming elements 715 and 718. According to anotherembodiment, virtual camera position data provided to the augmentedreality module by a surround view software module and pose estimationfor vehicle 705 are used to determine the position of gaming elements717 and 718.

According to one embodiment, input commands to rotate and/or translateone or more of vehicle avatar 716 and virtual camera position 700 aredetected by the augmented virtual vehicle view with surround viewimaging and as result can include updating of both stitched image dataand display characteristics of augmented elements.

FIG. 7 illustrates an exemplary representation of an updated virtualcamera position 720 depicted relative to a vehicle 705 with surroundview image 710, wherein direction 722 indicates the virtual cameraposition view and direction associated with a stitched image of updatedvirtual camera position 720. According to one embodiment, updatedvirtual camera position 720 also relates to an above the roof view ofvehicle 705. According to one embodiment, a graphical representation 721of the updated virtual camera position includes stitched image 725representing a portion of surround view image 710 associated withdirection 722. According to one embodiment, an augmented reality moduleaugments stitched image 725 to include vehicle avatar 716 as beingtranslated toward gaming element 718. Accordingly, a change in thevirtual camera position detected by a controller input module allows formanipulation such as rotation or translation of the stitched image viewand allows for virtual gaming with surround view image data.

FIG. 8 depicts a system diagram according to one or more embodiments.According to one embodiment, the augmented surround view systemsdiscussed herein can be provided for vehicle passengers. FIG. 8 depictssystem 800 servicing multiple display panels and controllers. In oneembodiment, system 800 allows for multiple passengers, such as bus, van,etc.

According to one embodiment, a vehicle system 805 includes one or moremodules to output augmented surround view image data. In one embodiment,vehicle system 805 interfaces with a plurality of displays 810 _(1-n)wherein each display 810 _(1-n) may be independently controlled by acontroller, such as controllers 815 _(1-n). According to one embodiment,each display 810 _(1-n) may allow for manipulation or adjustment of avirtual camera position relative to the camera, such that display 810 ₁may display one direction (e.g., direction 611), while display 810 ₂ maydisplay a stitched image according to another direction (e.g., direction622). According to another embodiment, each display may provide adifferent set of iconography or filters based on user selections.

While this disclosure has been particularly shown and described withreferences to exemplary embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the claimedembodiments.

What is claimed is:
 1. A method for presentation and control of anaugmented virtual vehicle view with surround view imaging, the methodcomprising: generating a stitched image view based on surround viewimage data for a vehicle; generating a pose estimation for the vehicle,wherein the pose estimation provides a vehicle position and a vehicleorientation relative to the stitched image view; augmenting the stitchedimage view to include one or more graphical elements based on the poseestimation for the vehicle; and presenting a graphical display of thestitched image view including the one or more graphical elements and avehicle avatar.
 2. The method of claim 1, wherein the surround viewimage data includes image data for a complete 360 degree circumferentialimage with respect to a position of the vehicle.
 3. The method of claim1, wherein the pose estimation for the vehicle is based on datagenerated by a vehicle sensor for acceleration and tilt of the vehicle.4. The method of claim 1, wherein augmenting the stitched image viewfurther includes augmenting the stitched image view to includeiconography for points of interest based on the pose estimation for thevehicle, and wherein the graphical display of the stitched image viewfurther includes the iconography for points of interest.
 5. The methodof claim 4, wherein the iconography representing the points of interestis enlarged as the vehicle nears one of the points of interest.
 6. Themethod of claim 4, wherein augmenting the stitched image to include theiconography for the points of interest includes augmenting the stitchedimage with graphical elements to match physical locations of elementsoutside of the vehicle.
 7. The method of claim 4, further comprisingdetecting position commands for a position of the vehicle avatar, andupdating the position of the vehicle avatar and the iconographyresponsive to detecting the position commands.
 8. A system of a vehiclefor presentation and control of an augmented virtual vehicle view withsurround view imaging, comprising: a display; multiple image sensors;and a vehicle control unit configured to: generate a stitched image viewbased on surround view image data for a vehicle received, the surroundview image data received via the plurality of image sensors, generate apose estimation for the vehicle, wherein the pose estimation provides avehicle position and a vehicle orientation relative to the stitchedimage view, augment the stitched image view to include one or moregraphical elements based on the pose estimation for the vehicle, andpresent a graphical display of the stitched image view including the oneor more graphical elements and a vehicle avatar via the display.
 9. Thesystem of claim 8, wherein the vehicle control unit is furtherconfigured to detect position commands for a position of the vehicleavatar within the graphical display and update a position of the vehicleavatar in the graphical display.
 10. The system of claim 8, wherein theone or more graphical elements include iconography, wherein the vehiclecontrol unit is configured to present the iconography for points ofinterest in the graphical display based on the pose estimation for thevehicle.
 11. The system of claim 8, wherein the vehicle control unit isfurther configured to modify the one or more graphical elements and theavatar as the vehicle moves, where the one or more elements includepoints of interest represented by iconography.
 12. The system of claim8, further comprising a vehicle sensor for detecting acceleration andtilt of the vehicle, and wherein the pose estimation for the vehicle isbased on data generated by the vehicle sensor for detecting accelerationand tilt of the vehicle.
 13. The system of claim 8, wherein thepresentation of the vehicle avatar is based on a virtual cameraposition, wherein the virtual camera position does not relate to anactual position one of the multiple image sensors.
 14. The system ofclaim 13, wherein the presentation of the one or more graphical elementsand the vehicle avatar is based on the virtual camera position.
 15. Amethod for presentation and control of an augmented virtual vehicle viewwith surround view imaging, the method comprising: generating a stitchedimage view based on surround view image data for a vehicle; generating apose estimation for the vehicle, wherein the pose estimation provides avehicle position and a vehicle orientation relative to the stitchedimage view; augmenting the stitched image view to include one or moregraphical elements based on the pose estimation for the vehicle;presenting a graphical display of the stitched image view including theone or more graphical elements and a vehicle avatar; and detectingposition commands for a position of the vehicle avatar within thegraphical display and updating the avatar position and iconography inresponse to the position commands.
 16. The method of claim 15, whereinpresentation of the graphical display of the stitched image viewincluding the one or more graphical elements includes arranging theiconography for one or more points of interest in a display window basedon the pose estimation.
 17. The method of claim 16, wherein the poseestimation and a virtual camera position allow for placement of theiconography into the stitched image based on the vehicle orientation.18. The method of claim 15, wherein updating the iconography in responseto the position commands includes increasing a size of the iconography.19. The method of claim 15, further comprising detecting changes to avirtual camera position and updating the graphical display to modify aposition of the one or more graphical elements.
 20. The method of claim15, wherein the surround view image data is generated via multiple imagesensors, and wherein the presentation of the vehicle avatar is based ona virtual camera position, wherein the virtual camera position does notrelate to an actual position one of the multiple image sensors.